Information processing device, information processing system and information processing method

ABSTRACT

An information processing device includes an acquisition unit which acquires information of acceleration, angular velocity, and magnetic intensity of an operation device, a coordinate conversion unit which converts the angular velocity acquired by the acquisition unit into global angular velocity in a global coordinate system using information of posture angles of the operation device, an initial posture angle calculation unit which calculates a posture angle of the operation device in the initial stage in the global coordinate system based on the information of the acceleration and the magnetic intensity, an updating unit which updates the posture angle of the operation device in the global coordinate system based on information of the global angular velocity, and a control unit which causes the coordinate conversion unit to convert the first angular velocity into the global angular velocity and to convert a second angular velocity into the global angular velocity.

BACKGROUND

The present disclosure relates to an information processing device, aninformation processing system, and an information processing methodwhich are used to execute calculation for realizing motions of anoperation target according to motions of an operation device thatincludes an acceleration sensor, and the like, based on informationobtained from the acceleration sensor, and the like, when a useroperates the operation device in a three-dimensional space.

In recent years, various kinds of space operation type operation devicesor input devices have been proposed. For example, an input devicedescribed in Japanese Unexamined Patent Application Publication(Translation of PCT Application) No. 6-501119 detects six-dimensionalmotions of a mouse in a three-dimensional space. Specifically, the inputdevice includes an accelerometer which detects acceleration inorthogonal three axes, and a rotation speed sensor which detectsrotation speed in the vicinity of the orthogonal three axes. A systemincluding the input device determines a position, a posture, and thelike of its own based on obtained accelerations and rotation speeds, anda computer such as a display device enables a GUI (Graphical UserInterface) to realize motions according to the motions of the inputdevice (for example, refer to Japanese Unexamined Patent ApplicationPublication (Translation of PCT Application) No. 6-501119).

SUMMARY

When a user operates the operation target using the operation devicesuch as the input device, or the like described in Japanese UnexaminedPatent Application Publication (Translation of PCT Application) No.6-501119, however, it is necessary to hold the operation device so thatthe detection axis of the sensor included in the operation devicecoincides with the axis of the operation target. This means that it isnecessary for the user to turn the operation device toward apredetermined direction while checking a part held in his or her handwhen the user starts an operation of holding the operation device, orwhen the user corrects an error (an error between motions of theoperation device and motions of the operation target, or the like).

As such, when the user frequently turns the eyes or the user frequentlypays attention to the holding part, such as checking his or her ownposition holding the operation device, it is easy for the user to haveautonomic dystonia such as fatigue, motion sickness, or the like.

Thus, it is desired to realize an operation device that makes theoperation target be operable even when the user orients toward anarbitrary direction, or holds the operation device in an arbitraryholding way, in other words, even when the operation device is held inan arbitrary posture in a three-dimensional space. In order to realizesuch an operation device, it is necessary to recognize the relationshipbetween the detection axis of a sensor and the axis of the operationtarget (in other words, the posture of the operation device in athree-dimensional space) not by consciousness of the user, but by theoperation device or a system including the device.

In addition, particularly when an operation device provided with anacceleration sensor is operated and moved, both gravity acceleration andmovement acceleration are detected by the acceleration sensor, andtherefore, research has to be conducted for calculation for recognitionof the posture of the operation device.

It is desirable to provide an information processing device, aninformation processing system, and an information processing methodwhich enables an operation by an operation device by recognizing theposture of the operation device while suppressing occurrence of acalculation error even when the posture of the operation device held bya user is any posture in a three-dimensional space.

An information processing device according to an embodiment of thedisclosure is equipped with an acquisition unit, a coordinate conversionunit, an initial posture angle calculation unit, an updating unit, and acontrol unit.

The operation device includes an acceleration sensor having anorthogonal three-axis detection axis, an angular velocity sensor havingthe orthogonal three-axis detection axis, and a magnetic sensor havingthe orthogonal three-axis detection axis in a local coordinate system,and can be operated in an arbitrary posture by a user in athree-dimensional space.

The acquisition unit acquires information of acceleration, angularvelocity, and magnetic intensity of an operation device which aredetected by each of the acceleration sensor, the angular velocitysensor, and the magnetic sensor thereof.

The coordinate conversion unit converts the angular velocity acquired bythe acquisition unit into a global angular velocity in a globalcoordinate system which indicates the three-dimensional space based oninformation of posture angles of the operation device in the globalcoordinate system.

The initial posture angle calculation unit calculates a posture angle ofthe operation device in the initial stage in the global coordinatesystem out of the posture angles based on the information of theacceleration and the magnetic intensity acquired by the acquisition unitat the start of an operation of the operation device by the user.

The updating unit updates the posture angle of the operation device inthe global coordinate system based on information of the global angularvelocity converted by the coordinate conversion unit.

The control unit causes the coordinate conversion unit to convert thefirst angular velocity acquired by the acquisition unit at the start ofthe operation into the global angular velocity using information of theinitial posture angle calculated by the angle calculation unit and toconvert a second angular velocity acquired after the acquisition of thefirst angular velocity into the global angular velocity usinginformation of the updated posture angle.

According to the embodiment of the disclosure, posture angles includingthe initial posture angle of the operation device in the globalcoordinate system are calculated based on information of acceleration,angular velocity, and magnetic intensity of the orthogonal three axes.Thus, the user can operate an operation target using the operationdevice in an arbitrary posture in the three-dimensional space.

In addition, when the operation is started by the user, the firstangular velocity is converted by the coordinate conversion unit usinginformation of the initial posture angle calculated based on informationincluding the acceleration obtained by the acceleration sensor. In otherwords, in the moment of starting the operation, the acceleration sensordetects only gravity acceleration in practice, the initial posture angleis calculated based on information including the gravity acceleration bythe initial posture angle calculation unit, and the coordinateconversion unit uses the calculated information.

However, during the operation by the user thereafter, the accelerationsensor detects a value obtained by adding movement acceleration (inertiaacceleration) to the gravity acceleration. Thus, if a posture angle iscalculated based on acceleration including such movement accelerationobtained by the acceleration sensor during the operation by the user, aserious error occurs.

Therefore, a coordinate conversion process is executed for the secondangular velocity acquired after the first angular velocity acquired asthe initial value based on information not including the information ofthe movement acceleration, that is, based on information of a postureangle updated at least once.

Accordingly, it is possible to suppress the circumstance in which theinfluence caused by the movement acceleration generated during theoperation of the operation device by the user is exerted duringcalculation of a posture angle of the operation device and therebycausing an error in the calculation.

The coordinate conversion unit may convert the acceleration acquired bythe acquisition unit into global acceleration in the global coordinatesystem based on information of a posture angle of the operation deviceupdated by the updating unit. Since the information of the updatedposture angle by the updating unit does not include the information ofthe movement acceleration as described above, it is possible to reflectmotions of the operation device on the operation target by convertingthe acceleration into the global acceleration based on the informationof the posture angle.

The information processing device may further include a determinationunit which determines that an operation of the operation device has beenstarted by the user using information of at least one of theacceleration, the angular velocity and the magnetic intensity acquiredby the acquisition unit. Accordingly, since the time of starting theoperation of the operation device is regulated, an output time point ofan initial posture angle of the operation device can be tightlyregulated.

The operation target of the operation device may be a three-dimensionalimage displayed on the screen of a display device which is electricallyconnected to the information processing device.

An information processing system according to another embodiment of thedisclosure includes an operation device and a control device, theoperation device includes an acceleration sensor having an orthogonalthree-axis detection axis, an angular velocity sensor having theorthogonal three-axis detection axis, a magnetic sensor having theorthogonal three-axis detection axis in a local coordinate system, and atransmission unit transmitting information of acceleration, angularvelocity, and magnetic intensity detected by each of the accelerationsensor, the angular velocity sensor, and the magnetic sensor, and can beoperated in an arbitrary posture by a user in a three-dimensional space,and the control device includes a reception unit which receives theinformation transmitted from the operation device, and a coordinateconversion unit, an initial posture angle calculation unit, an updatingunit, and a control unit as described above.

An information processing method according to still another embodimentincludes acquiring information of acceleration, angular velocity, andmagnetic intensity of an operation device, which includes anacceleration sensor having an orthogonal three-axis detection axis, anangular velocity sensor having the orthogonal three-axis detection axis,and a magnetic sensor having the orthogonal three-axis detection axis ina local coordinate system and can be operated in an arbitrary posture bya user in a three-dimensional space, which are detected by each of theacceleration sensor, the angular velocity sensor, and the magneticsensor, calculating a posture angle of the initial stage out of postureangles of the operation device in a global coordinate system indicatingthe three-dimensional space based on information of the acceleration andthe magnetic intensity acquired at the start of an operation of theoperation device by the user, converting a first angular velocityacquired at the start of the operation into global angular velocity inthe global coordinate system using the calculated posture angle of theinitial stage, calculating a new posture angle instead of the postureangle of the initial stage based on information of the converted globalangular velocity, and converting a second angular velocity acquiredafter the acquisition of the first angular velocity into the globalangular velocity using information of the calculated new posture angle.

According to the embodiments of the disclosure, even if the operationdevice held by the user takes any posture in the three-dimensionalspace, the posture of the operation device can be recognized whilesuppressing the occurrence of a calculation error, whereby an operationby the operation device is possible.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an information processing system accordingto an embodiment of the present disclosure;

FIG. 2 is a diagram showing a configuration of hardware of an operationdevice according to an embodiment of the disclosure;

FIG. 3 is a diagram showing a configuration of hardware of a displaydevice according to an embodiment of the disclosure;

FIG. 4 is diagram for illustrating detection axes of each sensor of theoperation device and a relative arrangement thereof;

FIG. 5 is a diagram showing the relationship between a local coordinatesystem and a global coordinate system;

FIG. 6 is a flowchart showing a process of the operation device;

FIG. 7 is a diagram for illustrating the principle of the calculationprocess shown in FIG. 6 and showing the state where the inclination(posture) of a sensor substrate is inclined in the global coordinatesystem;

FIG. 8 is a diagram for illustrating the principle of the calculationprocess shown in FIG. 6 as in FIG. 7;

FIG. 9 is a diagram showing the appearance when a user actually operatesan object using the operation device according to the embodiment of thedisclosure;

FIG. 10 is a diagram showing a distance between a display unit of adisplay device and (eyes) of a user;

FIG. 11 is a diagram for illustrating a display principle of 3D video ina depth direction (X axis direction) and shows a state where an objectis displayed inside a display unit; and

FIG. 12 is a diagram for illustrating a display principle of 3D video ina depth direction (X axis direction) and shows a state where an objectis displayed in front of the display unit.

DETAILED DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of the present disclosure will be describedwith reference to drawings.

[Configuration of Information Processing System]

FIG. 1 is a diagram showing an information processing system accordingto an embodiment of the disclosure.

The information processing system includes an operation device 10 whichis operated by a user, and a display device 50 which receives operationinformation transmitted from the operation device 10 to execute adisplay process based on the information. The operation device 10 andthe display device 50 are electrically connected to each other, and inthe embodiment, the units are connected so that wireless communicationis performed through infrared rays, radio waves, or the like.

The operation device 10 is formed in a hand-held size, and formed in,for example, a spherical shape. The display device 50 functions as acontrol device executing control of display such as rotating or movingan object (operation target) 51 in the screen of a display unit 52 withoperation of a user using the operation device 10.

As a typical example of the display device 50, there is, for example, a3D (dimension) TV, or the like, which can display the 3D object 51. Theoperation target is not limited to 3D images, but may be 2D images suchas icons, pointers, and the like. Alternatively, the icons and pointersmay be displayed as 3D images.

FIG. 2 is a diagram showing a configuration of hardware of the operationdevice 10 according to an embodiment of the disclosure. The operationdevice 10 includes an acceleration sensor 5, an angular velocity sensor6, a magnetic sensor 7, a CPU 2, a RAM 3, a RAM 4, and a transmitter 8.In addition, the operation device 10 includes a power source not shownin the drawing, a pressure sensor 9, a rewritable memory, and the like.Instead of the CPU 2, programmable hardware, for example, an FPGA (FieldProgrammable Gate Array), or the like may be used. As the angularvelocity sensor 6, a device is used, which detects an angular velocityusing, for example, a Coriolis force.

The hardware is arranged in a housing 11 of a spherical shape so as tobe fixed to the housing 11. The pressure sensor 9 is installed in pluralin, for example, the inner side of the housing 11, and the pressuresensor group detects a pressured position by a user and the pressuretherein.

FIG. 3 is a diagram showing a configuration of hardware of the displaydevice 50 according to an embodiment of the disclosure. The displaydevice 50 includes a CPU 53, a ROM 54, and a RAM 55 as in generalcomputers, and also includes a display unit 52, a communication unit 56,and a storage unit 57. The communication unit 56 mainly functions as areceiver here. The storage unit 57 is a typical auxiliary (secondary)storage unit of the ROM 54 and the RAM 55.

Furthermore, the display device 50 has a configuration in which acontrol part (control device) which receives information transmittedfrom the operation device 10 and controls the display of the object 51is integrated with the display unit 52, but the control part and thedisplay unit may be connected to each other as separate units so as toperform communication in a wire or wireless manner.

FIG. 4 is diagram for illustrating detection axes of each sensor of theoperation device 10 and a relative arrangement thereof. All of theacceleration sensor 5, the angular velocity sensor 6, and the magneticsensor 7 have orthogonal three-axis detection axes, respectively. Inother words, the acceleration sensor 5 has sensors (5 a, 5 b, and 5 c)corresponding to three detection axes. The angular velocity sensor 6 andthe magnetic sensor 7 also have sensors (6 a, 6 b, and 6 c), and (7 a, 7b, and 7 c), respectively.

For example, all the sensors 5, 6, and 7 are packaged in single sharedpackage. Alternatively, the sensors 5, 6, and 7 are packaged in separatepackages and mounted on single shared sensor substrate.

The ROM 4, a memory, which is not shown in the drawing, of the operationdevice 10, and/or the ROM 54 and the storage unit 57 of the displaydevice 50 store software, or the like, for realizing a process shown inFIG. 6 to be described later. The process shown in FIG. 6 is a processfor recognizing the posture of the operation device 10. In an embodimentto be described below, the CPU 2 and the above-described software mainlyfunction as control units.

FIG. 5 is a diagram showing the relationship between a local coordinatesystem and a global coordinate system.

The display device 50 is put on the ground surface. Herein, a coordinatesystem which is fixed to the ground surface or the display device 50 iscalled a global coordinate system. In addition, a coordinate systemwhich is fixed to the sensor substrate 20 of the operation device 10including the coordinate that can freely move with respect to the globalcoordinate system is called a local coordinate system. The sensorsubstrate 20 is a shared substrate on which the acceleration sensor 5,the angular velocity sensor 6, and the magnetic sensor 7 are mounted asdescribed above.

For convenience of description, hereinafter, the global coordinatesystem is indicated by large letters (X, Y, and Z), and the localcoordinate system is indicated by small letters (x, y, and z). However,in order to facilitate understating of sentences, description will beprovided using the expressions “local” and “global” as far as possible.The ground surface is set to an X-Y plane in the global coordinatesystem, and a surface parallel to the main surface of the substrate isset to an x-y plane in the local coordinate system.

[Process Content of Operation Device]

FIG. 6 is a flowchart showing a process of the operation device.

A signal output from each of the sensors 5 (5 a, 5 b, and 5 c), 6 (6 a,6 b, and 6 c), and 7 (7 a, 7 b, and 7 c) is converted to digitalinformation by an AD converter (Step ST101). In this case, the CPU 2mainly functions as an acquisition unit which acquires information ofacceleration, angular velocity, and magnetic intensity. Informationacquired by the acquisition unit is first used in a process of startdetermination (Step ST102). The start determination is determination ofwhether or not a user is holding the operation device and started anoperation.

The start determination process is executed using information obtainedfrom, for example, at least one of the acceleration sensor 5, theangular velocity sensor 6, and the magnetic sensor 7. In FIG. 6, thestart determination process is executed using information obtained from,for example, all of the sensors 5, 6 and 7, but a start determinationprocess using Formulas 1 and 2 below uses only the acceleration sensor5. When the start determination process is executed, for example, theCPU 5 and software in which the calculation content is describedfunction as determination units.√{square root over (a _(x) ² +a _(y) ² +a _(z) ²)}=Constant  Formula 1B<√{square root over (a _(x) ² +a _(y) ² +a _(z) ²)}<C  Formula 2

a_(x), a_(y), a_(z): acceleration of three axes (local acceleration)detected by the acceleration sensor (5 a, 5 b, and 5 c)

B, C: constant

The “constant” of Formula 1 is set to, for example, a value close tozero. When Formula 1 is not valid, it means that movement accelerationis added to the acceleration sensor 5 in addition to gravityacceleration, and it is determined that the operation device has beenmoved. In Formula 2, when the detected acceleration is out of apredetermined range, it is determined that the operation device has beenmoved.

Processes after Step ST103 are processes executed when it is determinedthat an operation has started.

When an operation by the operation device 10 has started, in Step ST103,initial posture angles around global X and Y axes of the operationdevice 10 are calculated respectively based on information ofacceleration (particularly, acceleration (a_(x), a_(y)) in the x and yaxis directions) detected by the acceleration sensor 5. FIG. 7 is adiagram for illustrating the principle of the calculation process andshowing the state where the inclination (posture) of the sensorsubstrate 20 is inclined in the global coordinate system.

When the initial posture angle around, for example, the global Y axis,that is an angle θ_(x) from the X axis is to be calculated, thecalculation is performed with Formula 3 below based on the informationof the acceleration a_(x). In Formula 3, as shown in FIG. 8, a value ofgravity 1G detected by the acceleration sensor 5 a of the x axis is setto, for example, A_(xG), a value of the acceleration sensor 5 a is setto a_(x) in a state where the sensor substrate 20 is inclined (a statewhere the acceleration sensor 5 (5 a) is inclined).

In the same manner, the initial posture angle around the global X axis,that is, an angle θ_(y) from the Y axis is calculated with Formula 4based on the information of the acceleration a_(y). A value of theacceleration sensor 5 b of the y axis for gravity 1G is set to A_(yG).θ_(x) =−a sin(a _(x) /A _(xG)) when a _(x)<0 and a _(z)>0θ_(x)=180+a sin(a _(x) /A _(xG)) when a _(x)<0 and a _(z)<0θ_(x)=180+a sin(a _(x) /A _(xG)) when a _(x)>0 and a _(z)<0θ_(x)=360+a sin(a _(x) /A _(xG)) when a _(x)>0 and a _(z)>0  Formula 3θ_(y) =−a sin(a _(y) /A _(yG)) when a _(y)<0 and a _(z)>0θ_(y)=180+a sin(a _(y) /A _(yG)) when a _(y)<0 and a _(z)<0θ_(y)=180+a sin(a _(y) /A _(yG)) when a _(y)>0 and a _(z)<0θ_(y)=360+a sin(a _(y) /A _(yG)) when a _(y)>0 and a _(z)>0  Formula 4

A_(xG), A_(yG): gravity acceleration 1G detected by each of theacceleration sensors 5 a and 5 b of the x and y axes

a_(x), a_(y), a_(z): the current value detected by each of theacceleration sensors 5 a, 5 b, and 5 c of the x, y, and z axes (localacceleration)

θ_(x), θ_(y): initial posture angle from the X and Y axes of the sensorsubstrate in the global coordinate system

Furthermore, a sine is used in Formulas 3 and 4, but the initial postureangle can be calculated also with a cosine or other calculation methods.

Next, in Step ST104, the initial posture angle (azimuth) around theglobal Z axis of the operation device 10 is calculated based oninformation calculated in Step ST103 and information of magneticintensities detected by the magnetic sensor 7 in each axis direction inthe local coordinate system.

For the calculation, Formula 5 below is used. In Formula 5, magneticintensities (local magnetic intensities) detected by each of themagnetic sensors 7 a, 7 b, and 7 c of the x, y, and z axes are set toh_(x), h_(y), and h_(z). In addition, magnetic intensities (globalmagnetic intensities) of the global X, Y, and Z axis directions obtainedfrom the calculation are set to H_(x), H_(y), and H_(z).H _(x) =h _(x) cos θ_(y) +h _(y) sin θ_(x) sin θ_(y) −h _(z) cos θ_(x)sin θ_(y)H _(y) =h _(y) cos θ_(x) +h _(z) sin θ_(x)θ_(z) (azimuth)=arctan(H _(y) /H _(x))  Formula 5

h_(x), h_(y), h_(z): magnetic intensity of the x, y, and z axisdirections in the local coordinate system (local magnetic intensity)

H_(x), H_(y): magnetic intensity of the global X and Y axis directions(global magnetic intensity)

θ_(z): initial posture angle around the global Z axis (azimuth)

The operation device 10 can recognize the azimuth of the sensorsubstrate 20 around the global Z axis by using the magnetic sensor 7which detects geomagnetism as above. When Formulas 3 to 5 arecalculated, the CPU 2 and software in which the calculation content isdescribed function as initial posture angle calculation units.

As above, the operation device 10 can recognize the initial posture(inclination for the X, Y, and Z axes) of the sensor substrate 20 in theglobal coordinate system with the calculation process of Steps ST103 andST104. In other words, the operation device 10 can recognize its ownposture without being conscious of the way and the direction of holdingthe operation device 10 by the user. As a result, the user can start anoperation using the operation device which takes an arbitrary posture ina three-dimensional space.

Next, in Step ST105, angular velocity detected by the angular velocitysensor 6 at the operation start, in other words, immediately after theoperation start, is converted into a global angular velocity in theglobal coordinate system based on information of the initial postureangles (θ_(x), θ_(y), and θ_(z)) calculated in Steps ST103 and ST104. Inother words, this is rotational coordinate conversion. The calculationprocess uses Formula 6. In this case, the CPU 2 and software in whichthe calculation content is described function as coordinate conversionunits.

$\begin{matrix}{\begin{pmatrix}W_{x} \\W_{y} \\W_{z}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\;\theta_{x}} & {\sin\;\theta_{x}} \\0 & {{- \sin}\;\theta_{x}} & {\cos\;\theta_{x}}\end{pmatrix}\begin{pmatrix}{\cos\;\theta_{y}} & 0 & {{- \sin}\;\theta_{y}} \\0 & 1 & 0 \\{\sin\;\theta_{y}} & 0 & {\cos\;\theta_{y}}\end{pmatrix}\begin{pmatrix}{\cos\;\theta_{z}} & {\sin\;\theta_{z}} & 0 \\{{- \sin}\;\theta_{z}} & {\cos\;\theta_{z}} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}w_{x} \\w_{y} \\w_{z}\end{pmatrix}}} & {{Formula}\mspace{14mu} 6}\end{matrix}$

w_(x), w_(y), w_(z): angular velocity around the local x, y, and z axes(local angular velocity)

W_(x), W_(y), W_(z): angular velocity around the global X, Y, and Z axes(global angular velocity)

In Step ST106, an angle R_(x) around the global X axis is calculatedusing the global angular velocity calculated as above. In other words, aglobal angle (of the X axis direction component) is calculated. Thecalculation process uses Formula 7. The calculation process usestrapezoidal integration. Angles R_(y) and R_(z) around the global Y andZ axes are also calculated in the same manner as R. The meaning of theprocess of Step ST106 is to change the initial posture angles (θ_(x),θ_(y), and θ_(z)) to new posture angles (R_(x), R_(y), and R_(z)), thatis, to update the posture angles. In this case, the CPU 2 and thesoftware in which the calculation content is described function asupdating units.

$\begin{matrix}{{R_{x}\left( t_{n} \right)} = \frac{\left( {{W_{x}\left( t_{n - 1} \right)} + {W_{x}\left( t_{n} \right)}} \right) \times \Delta\; t}{2}} & {{Formula}\mspace{14mu} 7}\end{matrix}$

R_(x): an angle around the X axis in the global coordinate system(global angle (global posture angle))

(t_(n)): a value obtained in n-th orderΔt=t _(n) −t _(n−1)

If the global angle is calculated as above, the information istransmitted (output) to the display device 50 by the transmitter 8 (StepST107). In the current point, the global angle is a global angle at theinitial time (that is, of the operation start time). The display device50 can receive the information and display the object 51 setting theinitial global angle as a posture angle at the start of motions of theobject 51. If the display device 50 receives information of the globalangle for the second time and thereafter, the display device 50 displaysthe object 51 taking a posture according to the global angle on thedisplay unit 52.

Furthermore, the global angle can be obtained by various integrationprocesses such as the midpoint rule or the Simpson's rule, not beinglimited to the trapezoidal integration as in Formula 7.

Herein, in the initial stage, the coordinate conversion units executesthe coordinate conversion process for the local angular velocity(initial value) based on the information of the initial posture angles(θ_(x), θ_(y), and θ_(z)) in Step ST105. However, the coordinateconversion units execute the coordinate conversion process for the localangular velocity (value of the second time and thereafter) based on theinformation of the global angles (R_(x), R_(y), and R_(z)) calculated inStep ST106 in the process of Step ST105 of the second time andthereafter (other than the initial stage). The calculation process usesFormula 8 below.

$\begin{matrix}{\begin{pmatrix}W_{x} \\W_{y} \\W_{z}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\; R_{x}^{\prime}} & {\sin\; R_{x}^{\prime}} \\0 & {{- \sin}\; R_{x}^{\prime}} & {\cos\; R_{x}^{\prime}}\end{pmatrix}\begin{pmatrix}{\cos\; R_{y}^{\prime}} & 0 & {{- \sin}\; R_{y}^{\prime}} \\0 & 1 & 0 \\{\sin\; R_{y}^{\prime}} & 0 & {\cos\; R_{y}^{\prime}}\end{pmatrix}\begin{pmatrix}{\cos\; R_{z}^{\prime}} & {\sin\; R_{z}^{\prime}} & 0 \\{{- \sin}\; R_{z}^{\prime}} & {\cos\; R_{z}^{\prime}} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}w_{x} \\w_{y} \\w_{z}\end{pmatrix}}} & {{Formula}\mspace{14mu} 8}\end{matrix}$

The global angles of the operation device 10 gradually changes (everysecond). Therefore, specifically in Formula 8, the rotational coordinateconversion is performed based on a value sequentially with the globalangles added as shown in formulas below.R _(x) ′=R _(x)(t _(n))+R _(x)(t _(n+1))R _(y) ′=R _(y)(t _(n))+R _(y)(t _(n+1))R _(z) ′=R _(z)(t _(n))+R _(z)(t _(n+1))However, depending on the specification of software for displayingimages by the display device 50, (R_(x), R_(y), and R_(z)) may be usedinstead of (R_(x)′, R_(y)′, and R_(z)′) in Formula 8 (the same as inFormula 9 below).

The information of the global angles (R_(x), R_(y), and R_(z))calculated in Step ST106 is also used in the process of Step ST108. InStep ST108, the coordinate conversion units convert local accelerationinto global acceleration using the information of the global angles inthe same manner as in Formula 8. The calculation process uses Formula 9below.

$\begin{matrix}{\begin{pmatrix}A_{x} \\A_{y} \\A_{z}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 \\0 & {\cos\; R_{x}^{\prime}} & {\sin\; R_{x}^{\prime}} \\0 & {{- \sin}\; R_{x}^{\prime}} & {\cos\; R_{x}^{\prime}}\end{pmatrix}\begin{pmatrix}{\cos\; R_{y}^{\prime}} & 0 & {{- \sin}\; R_{y}^{\prime}} \\0 & 1 & 0 \\{\sin\; R_{y}^{\prime}} & 0 & {\cos\; R_{y}^{\prime}}\end{pmatrix}\begin{pmatrix}{\cos\; R_{z}^{\prime}} & {\sin\; R_{z}^{\prime}} & 0 \\{{- \sin}\; R_{z}^{\prime}} & {\cos\; R_{z}^{\prime}} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}a_{x} \\a_{y} \\a_{z}\end{pmatrix}}} & {{Formula}\mspace{14mu} 9}\end{matrix}$

A_(x), A_(y), A_(z): acceleration around the global X, Y, and Z (globalacceleration)

The coordinate conversion units execute coordinate conversion based onthe information of global angles in stages other than the initial stagealso in Step ST108 as above.

As above, if the initial posture angles are calculated for the firsttime, the global angles calculated in Step ST106 are used incalculations of the second time and thereafter without using the initialposture angles. Hereinbelow, the reason will be described.

At the start of the operation of the operation device 10 by the user,local angular velocity is converted into global angular velocity usingthe information of initial posture angles calculated based oninformation including acceleration obtained by the acceleration sensor5. In other words, at the moment of starting the operation, theacceleration sensor 5 detects only gravity acceleration practically;initial posture angles are calculated based on information including thegravity acceleration, and coordinate conversion is performed based onthe initial posture angles.

However, during the operation by the user thereafter, the accelerationsensor 5 detects a value obtained by adding movement acceleration(inertia acceleration) to the gravity acceleration. In other words,during the operation by the user, the posture of the sensor substrate 20changes every second. Thus, if a posture angle is calculated based onacceleration including such movement acceleration obtained by theacceleration sensor 5 during the operation of the user, a serious erroroccurs.

Therefore, in the present embodiment, a conversion process is executedfor local angular velocity acquired after local angular velocityacquired as an initial value based on information not includinginformation of movement acceleration, that is, information of a postureangle updated using local angular velocity, at least once (informationobtained after passing through Steps ST105 and ST106 at least once).This is because a value of the local angular velocity is not affected bymovement acceleration.

According to the calculation process, it is possible to suppress thecircumstance in which the influence caused by movement accelerationgenerated during the operation of the operation device 10 by the user isexerted during calculation of a posture angle of the operation device 10and thereby causing an error in the calculation.

Next, in Step ST109, velocity is calculated based on the globalacceleration calculated in Step ST108, and in Step ST110, a movementdistance is calculated from the velocity. The calculation process can beexecuted using various integration methods of the above-described StepST106. In Step ST111, the operation device 10 transmits (outputs) theinformation of the calculated distance to the display device 50. Thedisplay device 50 receives the distance information, and displays theobject 51, which moves as far as the distance, on the display unit 52.

As described above, since the information of the posture angle updatedin Step ST106 does not include information of movement acceleration, itis possible to faithfully reproduce the motions of the operation device10 as the motions of the object 51 by converting the local accelerationinto the global acceleration based on the information of the postureangle. Accordingly, the user can perform the operation intuitively.

As described above, in the present embodiment, even if the operationdevice 10 held by the user takes any posture in a three-dimensionalspace, it is possible that the posture of the sensor substrate 20 isrecognized while suppressing the occurrence of a calculation error,whereby the object 51 can be operated by the operation device 10.

In the above description, an embodiment in which the operation device 10executes all the processes shown in FIG. 6 is introduced. However, apart of the processes shown in FIG. 6 may be executed by the displaydevice 50. For example, the operation device 10 may execute the processup to Step ST101, and transmit information of acceleration, angularvelocity, and magnetic intensity detected by each of the sensors 5, 6,and 7 to the display device 50. The display device 50 receives theinformation, and executes the processes of Step ST102 and thereafter.Alternatively, the operation device 10 may execute the processes, forexample, up to Step ST102, . . . , or ST105, and the display device 50may execute the remaining processes.

Of which process the operation device 10 and the display device 50 willtake charge may be appropriately determined based on environmentalcomponents, for example, calculation process performance thereof, coststhereof, chip size thereof, and the like.

SPECIFIC DESCRIPTION OF APPLICATION EXAMPLE OF INFORMATION PROCESSINGSYSTEM

FIG. 9 is a diagram showing the appearance when a user actually operatesthe object 51 using the operation device 10 according to the embodimentof the disclosure. This is the appearance in which the user draws theoperation device 10 from the display unit 52 of a 3D TV as the displaydevice 50 along the negative direction of the X axis of the globalcoordinate system. In this case, the display device 50 displays as ifthe object 51 moves to the front of the user according to the operationof the user by adjusting parallax. In the same manner, when the userpushes the operation device 10 along the positive direction of the Xaxis of the global coordinate system, the display device 50 displays asif the object 51 moves in the depth direction of the display unit 52.

The three-dimensional display on the global X axis as above is possibleby associating the motions of the operation device 10 in the globalcoordinate with the parallax of video displayed on the display device50. As shown in FIG. 10, the distance between the display unit 52 of thedisplay device 50 and (eyes of) the user, that is, the viewing distanceis set to L.

FIGS. 11 and 12 are diagrams for illustrating a display principle of 3Dvideo in the depth direction (X axis direction). Referring to FIG. 11,if setting is performed as below:

L: viewing distance;

P: parallax of video;

E: distance between eyeballs; and

D: depth of the object, then, the following is valid,P=(D×E)/(D+L) based on D:P=(D+L):E.

For example, when E=65 mm, L=2 m, and the object 51 is viewed by theuser so as to be displayed at the position of 24 m (D=24 m) of the depthto the display unit 52, P=6 cm. In other words, it is better to generateparallax images 51 a and 51 b which are deviated from each other by 6cm, constituting the object 51.

The motions of the operation device 10 in the X axis direction of theglobal coordinate can be linked to 3D video by associating the parallaxP and a movement distance T of the operation device 10 in the screen inthe global coordinate system as P=αT (α is a proportional constant).

The relation shown in FIG. 12 takes the same point of view as in FIG.11.P=(D×E)/(L−D) based on D:P=(L−D):E

For example, when E=65 mm, L=2 m, and the object 51 is viewed by theuser so as to be displayed at the position of 1.2 m (D=1.2 m) of thefront from the display unit 52, P=10 cm. In other words, it is better togenerate parallax images 51 a and 51 b which are deviated from eachother by 10 cm, constituting the object 51.

The motions of the operation device 10 in the X axis direction of theglobal coordinate can be linked to 3D video by associating the parallaxP and a movement distance T of the operation device 10 in the screen inthe global coordinate system as P=αT (α is a proportional constant).

OTHER EMBODIMENT

An embodiment according to the disclosure is not limited to theembodiment described above, and realized in various forms ofembodiments.

In the above-described embodiment, a value detected by each of thesensors for the first time after the start of an operation is determinedat the start determination process of Step ST102 is described as a localvalue (local acceleration, local angular velocity, and local magneticintensity). Any one of values detected by each of the sensors for a fewpredetermined times from the time point when the start of the operationis determined at the start determination process, or the average value,the intermediate value, or the like of the values may be set to a localvalue. In other words, the start of the operation is a notion includinga predetermined time period after the operation is started, and a notionincluding a certain period to the extent that a user does not feel asense of incompatibility with the first movement of the object. Thepredetermined period is also based on a clock frequency of CPUs in theoperation device and the display device.

In the above-described embodiment, the operation device 10 and thedisplay device 50 are wirelessly connected, but may be connected to eachother in a wired manner. A communication medium between the devices maybe the Internet.

In the above-described embodiment, the case is exemplified in which theoperation device 10 and the display device 50 are in the same room, butfor example, the devices may be located in separate buildings, or remoteplaces. In that case, a communication medium between the devices may bethe Internet as described above.

The operation target may be a robot or other machine in the actualworld, not being limited to images displayed on a screen. In that case,a remote operation may be realized as above.

The operation device 10 described above has a spherical outer shape, butis not limited thereto, and can adopt various outer shapes.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2010-278809 filed in theJapan Patent Office on Dec. 15, 2010, the entire contents of which arehereby incorporated by reference.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

What is claimed is:
 1. An information processing device comprising: anacquisition unit configured to acquire information of acceleration,angular velocity, and magnetic intensity of an operation device, theoperation device comprising an acceleration sensor having an orthogonalthree-axis detection axis, an angular velocity sensor having theorthogonal three-axis detection axis, and a magnetic sensor having theorthogonal three-axis detection axis in a local coordinate system, theoperation device being configured for operation in an arbitrary postureby a user in a three-dimensional space, the acquisition unit beingconfigured to acquire acceleration information detected by theacceleration sensor, first angular velocity information detected by theangular velocity sensor, and magnetic intensity information detected bythe magnetic sensor; a coordinate conversion unit configured to convertthe angular velocity acquired by the acquisition unit into globalangular velocity in a global coordinate system which indicates thethree-dimensional space using information of posture angles of theoperation device in the global coordinate system; an initial postureangle calculation unit configured to calculate a posture angle of theoperation device in an initial stage in the global coordinate system outof the posture angles based at least in part on acceleration informationand magnetic intensity information acquired by the acquisition unit at astart of an operation of the operation device by the user, theacceleration information consisting of gravitational accelerationinformation and not inertial acceleration information, the magneticintensity information comprising magnetic intensities detected by themagnetic sensor in each of the three axis directions in the localcoordinate system; an updating unit configured to update the postureangle of the operation device in the global coordinate system based atleast in part on information of the global angular velocity converted bythe coordinate conversion unit; and a control unit configured to causethe coordinate conversion unit to convert the first angular velocityacquired by the acquisition unit at the start of the operation into theglobal angular velocity using information of the initial posture anglecalculated by the initial posture angle calculation unit, and to converta second angular velocity acquired after the acquisition of the firstangular velocity into the global angular velocity using information ofthe updated posture angle, and not information of inertial accelerationof the operation device occurring between the start of the operation andthe acquisition of the second angular velocity.
 2. The informationprocessing device according to claim 1, wherein the coordinateconversion unit converts the acceleration acquired by the acquisitionunit into global acceleration in the global coordinate system based atleast in part on information of a posture angle of the operation deviceupdated by the updating unit.
 3. The information processing deviceaccording to claim 1, further comprising: a determination unitconfigured to determine that an operation of the operation device hasbeen started by the user using information of at least one of theacceleration, the angular velocity and the magnetic intensity acquiredby the acquisition unit.
 4. The information processing device accordingto claim 1, wherein an operation target of the operation device is athree-dimensional image displayed on the screen of a display devicewhich is electrically connected to the information processing device. 5.An information processing system comprising: an operation device whichincludes an acceleration sensor having an orthogonal three-axisdetection axis, an angular velocity sensor having the orthogonalthree-axis detection axis, a magnetic sensor having the orthogonalthree-axis detection axis in a local coordinate system, and atransmission unit configured to transmit information of acceleration,angular velocity, and magnetic intensity detected by the accelerationsensor, the angular velocity sensor, and the magnetic sensor,respectively, the operation unit being configured for operation in anarbitrary posture by a user in a three-dimensional space; and a controldevice which includes: a reception unit configured to receive theinformation transmitted from the operation device, a coordinateconversion unit configured to convert a first angular velocity receivedfrom the operation device into global angular velocity in a globalcoordinate system using information of posture angles of the operationdevice in the global coordinate system indicating the three-dimensionalspace, an initial posture angle calculation unit configured to calculatea posture angle of the operation device, in an initial stage in theglobal coordinate system of the operation device out of the postureangles based at least in part on acceleration information and magneticintensity information received by the reception unit at a start of anoperation of the operation device by the user, the accelerationinformation consisting of gravitational acceleration information and notinertial acceleration information, the magnetic intensity informationcomprising magnetic intensities detected by the magnetic sensor in eachof the three axis directions in the local coordinate system, an updatingunit configured to update the posture angle of the operation device inthe global coordinate system based at least in part on information ofthe global angular velocity converted by the coordinate conversion unit,and a control unit configured to cause the coordinate conversion unit toconvert the first angular velocity received by the reception unit at thestart of the operation into the global angular velocity usinginformation of the initial posture angle calculated by the initialposture angle calculation unit and to convert a second angular velocityreceived after the reception of the first angular velocity into theglobal angular velocity using information of the updated posture angle,and not information of inertial acceleration of the operation deviceoccurring between the start of the operation and the acquisition of thesecond angular velocity.
 6. An information processing method comprisingacts of: acquiring information of acceleration, angular velocity, andmagnetic intensity of an operation device, which includes anacceleration sensor having an orthogonal three-axis detection axis, anangular velocity sensor having the orthogonal three-axis detection axis,and a magnetic sensor having the orthogonal three-axis detection axis ina local coordinate system, the operation device being configured foroperation in an arbitrary posture by a user in a three-dimensionalspace, the acceleration information being detected by the accelerationsensor, the angular velocity information being detected by the angularvelocity sensor, and the magnetic intensity information being detectedby the magnetic sensor; calculating a posture angle of an initial stageout of posture angles of the operation device in a global coordinatesystem indicating the three-dimensional space based at least in part onacceleration information and magnetic intensity information acquired ata start of an operation of the operation device by the user, theacceleration information consisting of gravitational accelerationinformation and not inertial acceleration information, the magneticintensity information comprising magnetic intensities detected by themagnetic sensor in each of the three axis directions in the localcoordinate system; converting a first angular velocity acquired at thestart of the operation into a global angular velocity in the globalcoordinate system using the calculated posture angle of the initialstage; calculating a new posture angle instead of the posture angle ofthe initial stage based at least in part on information of the convertedglobal angular velocity; and converting a second angular velocityacquired after the acquisition of the first angular velocity into theglobal angular velocity using information of the calculated new postureangle, and not information of inertial acceleration of the operationdevice occurring between the start of the operation and the acquisitionof the second angular velocity.